Motivation and Context

The chat completion agent was not emitting token using during streaming invocation because we were only allowing through response.items. In the case of token usage, response.items is [] and the usage is contained as part of the message's metadata dict. This PR fixes that bug and allows for response.items or response.metadata.get("usage"). Two new samples are added to the concepts/agents/chat_completion dir to show how one can track token use for streaming and non-streaming agent invocation. Token usage handling is also added to the chat completion agent integration tests.

Description

• Fixes a bug where we weren't emitting the streaming token usage for the chat completion agent. Also now includes the prompt_tokens_details and completion_tokens_details models that are returned, but not previously handled.
• Adds new samples
• Updates integration tests to track token usage and make sure they're non-zero.
• Closes Python: Bug: invoke_stream method in chat_completion_agent.py not returning token usage data #12411

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄